Koalas: Implement Koalas Missing APIs

[xinrong-databricks]

Help wanted! A few popular pandas APIs are missing in Koalas. We are going to implement them!

Please use this thread to comment on which function you will be working so we don't duplicate work. Please mention this issue in your PR so that the list below can be updated.

• DataFrame

  • [ ] [cov](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cov.html)
  • [x] [first](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.first.html)
  • [x] [itertuples](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.itertuples.html)
  • [ ] [resample](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html)
  • [x] [combine_first](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.combine_first.html)
  • [x] [between_time](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.between_time.html)
  • [ ] [boxplot](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.boxplot.html)
  • [x] [insert](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.insert.html)
  • [x] [swapaxes](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.swapaxes.html)

• Series

  • [ ] [ewm](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.ewm.html)
  • [ ] [resample](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.resample.html)
  • [x] [factorize](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.factorize.html)
  • [x] [last](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.last.html)
  • [x] [swapaxes](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.swapaxes.html)

• Index

  • [x] [to_list](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.to_list.html) / tolist
  • [ ] [map](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.map.html)

• SeriesGroupBy and DataFrameGroupBy

  • [x] [median](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.GroupBy.median.html)
  • [x] [tail](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.GroupBy.tail.html)

[AishwaryaKalloli]

Hi, I would like to help. I was planning on picking combine_first functionality.
Can you let me know if I can work on it. Thanks!

[HyukjinKwon]

Please go ahead @AishwaryaKalloli!

[xinrong-databricks]

Certainly, thank you @AishwaryaKalloli!

[AishwaryaKalloli]

Just finished the set up in my local, hopefully will have some updates soon!

[AishwaryaKalloli]

I have committed the code, let me know if it is in the right direction. If it is I'll add the test cases and docs.

[shril]

Hi @ueshin, @HyukjinKwon can I proceed with the following Dataframe APIs -

• cov
• first
• between_time
• insert

I'll start with the dev once you give me the approval.

[ueshin]

@shril sure, please go ahead! Thanks!

[shril]

@ueshin I was going through this blog post of yours - https://databricks.com/blog/2020/03/31/10-minutes-from-pandas-to-koalas-on-apache-spark.html, and it suggested using map_in_pandas as a better alternative workaround because it does not require moving data into a single client node and potentially causing out-of-memory errors.

Do you suggest to proceed with using the map_in_pandas() API?

Edit:
I found out that map_in_pandas() is getting deprecated.
Do you suggest to continue with apply_batch() API?

[ueshin]

@shril I don't have a strong opinion on it. If you can implement it without apply_batch(), you don't need to use it.

[shril]

Hi @ueshin, I am slightly confused. We don't have DatetimeIndex like pandas, so can we convert our index to pandas and use it's indexer_between_time function.

i = pd.date_range('2018-04-09', periods=2000, freq='1D1min')
ts = ks.DataFrame({'A': ['timestamp']}, index=i)
indexer = ts.index.to_pandas().indexer_between_time(start_time='0:15', end_time='0:45')
result = ts.copy().take(indexer)

This is the small implementation I tried. Do you think that to_pandas() might result in out-of-memory errors?

[chogg]

Is iterating through groups on the roadmap for API coverage? I would find that helpful.

[xinrong-databricks]

@chogg Thanks for the suggestion! We'll look into this and keep you updated.

[awdavidson]

Hi all, I was going to look at implementing functionality for last Is anyone looking at this?

[xinrong-databricks]

@awdavidson Certainly, please feel free to do so! Your PR seemed to be closed.

[awdavidson]

@xinrong-databricks I'll reopen currently still working on it. Opened the PR to check build etc as had issue running a few things locally - didn't want to clutter your PR tab. Local environment is now working so should be able to completely test :)

[xinrong-databricks]

@awdavidson Thanks! You might mark it as a draft PR until it's ready for review. Let us know if you have any questions :)

[awdavidson]

@ueshin @xinrong-databricks has there been any discussion around how to implement Index.map? I have a few idea's that maybe useful, however, do not want to step on anyones toes! If you have any information or documentation that would be useful! :)