Koalas: Does spark DataFrame.limit guarantees any order?

[guyao]

This is just a question, not an issue.
Out of curiosity, does spark.DataFrame.limit guarantees the order ?

That method is used to implement many methods like .head, .iloc
In pandas the data is small and the order could be guaranteed. But in koalas, the data is distributed. Is it guaranteed those methods would keep order?

[HyukjinKwon]

Yeah, this is a pretty important issue in Koalas.

Currently, it does not guarantees the natural order in general. Natural order can be kept when it only performs map operation (without shuffle) but if shuffle happens, the order cannot be kept.

So, if users want the order, they should sort by, for example, index.

[HyukjinKwon]

We should still think about how we'll handle this in general.

[guyao]

Thank you!

Follow up of my last question, do you mean spark DataFrame.limit(n) will return first n result ( in natural order, if no shuffle happens)

For example, if read a file as DataFrame, the file would be split into partitions P1, P2 ... , where P1 contains first part of file, ... If I tried to call df.limit(n), it will return first n of P1 (if n < # P1)

Is that correct? (sorry it more like a spark question but I asked here)

[HyukjinKwon]

Yes, that's correct :-).

[guyao]

I was thinking about what pandas users expected when using iloc[slice(start, end, step)]
Do people really know which start index, end index he need when he is using an uncountable dataframe.

( but I think user still expected order if the DF is small )

[HyukjinKwon]

Yes, it's a bit questionable at this moment. Maybe we should do one of these: ...

1. We just disallow all such cases
2. Document the case when the order can be deterministic
3. Document that the index always should be sorted before such operations for order guarantee
4. Internally always sort on the index before such operations in such cases