Koalas: Implement the first batch of Serialization / IO / Conversion functions

[rxin]

See https://pandas.pydata.org/pandas-docs/stable/reference/frame.html#serialization-io-conversion

Looks like we can easily implement almost all of them by calling toPandas().func_name().

One thing is that some of the functions support max_rows. When that argument is specified, we should add a limit call in Spark to avoid moving all the data to the driver.

The list to add in the first batch are:

• [x] to_dict (see #169)
• [x] to_excel (#288)
• [x] to_html (we already have this, but let's add a limit when max_rows is set), done in #206
• [x] to_latex (#297)
• [x] to_records (#298)
• [x] to_string (done in #211 and #213)
• [x] to_clipboard (#257)

Skipping the following because I don't know how popular they are:

• to_pickle
• to_hdf
• to_stata
• to_msgpack
• to_records
• to_sparse
• to_dense

The following might require parallelization with Pandas UDFs, rather than collecting all the data to the driver, so leaving them for the future:

• to_sql
• to_gbq

I'm also not adding json and csv here. We need to design those properly because both Spark and Pandas have those.

[rxin]

For contributors -- in general this is a pretty easy one to get started. Just look at how we did https://github.com/databricks/koalas/pull/211 and https://github.com/databricks/koalas/pull/213

[arjun180]

Can I work on this?

[HyukjinKwon]

please go ahead.

[shril]

@rxin , @HyukjinKwon Can I take this up as well?

[rxin]

Go for it.

On Thu, May 2, 2019 at 12:47 PM Shril Kumar [e-mail]
wrote:

@rxin https://github.com/rxin , @HyukjinKwon
https://github.com/HyukjinKwon Can I take this up as well?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/databricks/koalas/issues/169#issuecomment-488807335,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACO6PA3QJ6XEWUOQMVLMHTPTNAOLANCNFSM4HIHAYQQ
.

[HyukjinKwon]

I think we should do to_json and to_csv too.

[garawalid]

@rxin

I'm also not adding json and csv here. We need to design those properly because both Spark and Pandas have those.

I agree with @HyukjinKwon, we can add to_json & to_csv.
In fact, both methods need to collect data in the driver through toPandas(). And if we pass the path as a parameter, the output will be saved in the driver. This is what Pandas users will expect, at least, those who are not familiar with Spark.

In the case of Spark, I think we should keep to_csv and to_json in a separate section than Serialization / IO / Conversion and we explain the behaviour in the documentation. These methods will be useful if we want to write the output in HDFS without impacting the performance.

Btw, to_clipboard uses to_csv.

[HyukjinKwon]

We can support both hdfs and local path and that's what Spark currently supports. Let's keep it consistent first before we start to have some divergence.

We can explain the difference in the docstring with notes and I think that's already what are doing.

[rxin]

@garawalid do you want to do the rest as well, so we can close this ticket?

[garawalid]

@rxin
Of course, I'll handle the rest!
You can close it.

[rxin]

We only close the ticket once all the items are done :)

[shril]

@garawalid are you doing to_excel function?

[garawalid]

@shril not yet! You can pick it up.

[abishekganesh72]

I can take up one of the functions. Let me know if something is available.

[HyukjinKwon]

Please say that you're working on some functions here and go ahead.

[athena15]

Working on .to_latex() function. Pull request coming soon.

[abishekganesh72]

Working on to_records

[rxin]

Thank you all!

[shril]

@rxin, I have implemented to_records in PR #298.
Can you please close this issue?

[HyukjinKwon]

Thanks, @shril. This issue is nicely finished.