Koalas: DataFrame.cache()

[rxin]

This would be a super useful function to have, even though it's Spark specific. In terms of documentation, we should create a Spark specific section in frame.rst.

[thunterdb]

Unlike .select which exists in pandas, this function is very specific to spark. How about putting all the spark specific functions under a .spark accessor?

It looks like dask is not doing that and simply mixes pandas and dask-specific functions, but I am not sure this helps users:
http://docs.dask.org/en/latest/dataframe-api.html#store-dataframes

[shril]

@rxin, @thunterdb I am taking this up.

[HyukjinKwon]

hey @shril, have you made some progresses on this?

[thunterdb]

+1, this is very useful in practice.

Regarding the API, I am thinking that we can add this as a function in the databricks.koalas namespace instead of as a method to DataFrame. This way, users can write code that works for both pandas and spark dataframes, which helps with writing tests and with transitioning smoothly between koalas and pandas. Then the following code would work with both pandas or koalas:

df = ... # pandas or koalas dataframe
df = ks.cache(df)
# Alternative when chaining:
df = df.pipe(ks.cache)

Cons:

• not as discoverable by the user (will not find it with tab autocomplete)

Also, because caching can consumed resources and should be usually limited in scope, we could implement python guards:

with ks.cache(df):
    mi = df.min()
    ma = df.max()
# Will automatically uncache the dataframe at the end of the block.

See an explanation on how to implement here:
http://effbot.org/zone/python-with-statement.htm

[rxin]

Why not both? Discoverability is essential. The other thing is I don't think "you should be able to run exactly the same code in pandas by changing a line of import" is a valid goal. As soon as you merge anything that does approximate count, you will break that goal. So there is really no point in keeping that goal.

[shril]

@HyukjinKwon I have made some progress. I will update by tonight.
@thunterdb thanks for the input.

[shril]

@rxin, I'll submit 2 different PRs for this issue

• cache implementation in DataFrame.
• cache function in databricks.koalas namespace.

I am doing the former one first.

@thunterdb, I am using this - (Link) for Context Management/Python Guards.
I hope you find this ok. Please, reply if you want me to change this implementation.

Also, I have a silly doubt that when we will cache a koalas dataframe, are we going to yield a spark dataframe or a koalas dataframe?
In the PR, I have yielded a Spark DataFrame.

[HyukjinKwon]

Oh wait, let's just have one API rather then in both places of DataFrame and namespace.

We can do:

with df.cache() as df:
    df.blahblah()

and

df = df.cache()

@thunterdb and @rxin, this idea actually was proposed in PySpark side before as well (https://issues.apache.org/jira/browse/SPARK-16921). We can match the idea in case somebody brings it in to PySpark side later.

[garawalid]

I find it tricky to implement only cache() without giving access to configure the executor memory and other parameters like spark.memory.storageFraction.

We are making the assumption that the DataFrame will fit always the memory.

Any thoughts?

[shril]

@garawalid I am just persisting the corresponding Spark DataFrame of the Koalas DataFrame on which df.cache() is called.

Isn't it true that we have to take care of the DataFrame fitting in the memory when Pandas functions are called? Please correct me if I am wrong.

[garawalid]

@shril, in fact, we need to take care of the memory when we persist the data through cache().

[shril]

@garawalid, I have changed the implementation. Please have a look.