Koalas: pandas.DataFrame.tail

[tdliu]

Head is implemented but tail is not. Tail is not just useful for exploring data but also a very useful way of slicing data.

Example: users have data ordered by date. They want to take the last 30 days. They can just call tail(30)

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tail.html

[rxin]

Is the most common case people run tail after sorting? It's actually somewhat poor performance to implement a general purpose tail.

[ustcscgyer]

It is not as straightforward as it seems. In Pandas, all the data is already in memory, tail is not too much different to head. In SparkDataFrame, data is calculated at the time of consuming, getting tail means you have to materialize the whole data set.

[thunterdb]

spark tends to think of tail as what happened in a linked list (dropping just a few elements), but it is effectively used like .head or .limit on the end of the data.

I have started a small design doc on how we could support these use cases and at the same time ensure that the output is exactly the same as pandas despite the distributed nature of spark. I will share it this weekend or next week.

[rodrigue89]

If you run a large data set in pandas, it may be a good idea to run both head and tail methods to confirm the integrity of your data while uploading or after processing your data.

[brookewenig]

Any update on this? For the demo, I'd actually really like tail to be implemented.

[HyukjinKwon]

^ WDYT @rxin, @thunterdb and @ueshin?