Koalas: Add .iloc support

[brookewenig]

Can we add .iloc support please?

See this notebook. Doesn't throw a non-supported error, but rather 'DataFrame' object has no attribute 'iloc'.

[HyukjinKwon]

See this notebook. Doesn't throw a non-supported error, but rather 'DataFrame' object has no attribute 'iloc'.

this is fixed as of https://github.com/databricks/koalas/commit/71fa23a2d59d9962ceb4f5eb80d1f44d765a071a

[thunterdb]

Note that the patch referred to by @HyukjinKwon does not include a working .iloc, just a better error message.

@brookewenig can you tell us more about which behaviour of .iloc you want? Since Koalas/Spark does not have an ordering of the rows, I am thinking of the following:

• but a big warning in the doc that the outcome may differ because Spark has no intrinsic notion of ordering of the rows, and that .loc with indexing is preferred in most cases
• support some specific use cases:
  
  
  
  • .iloc[:10] -> .head(10) is the only one I have seen often in the wild.
  
  
  • .iloc[::, [0,1]] for getting a few columns is another one
  
  

[brookewenig]

@thunterdb I do not want my code to break when going from Pandas -> Koalas.

This is why I think we should have a guide on "gotchas" when going from single node to distributed. A pandas user might not understand that Spark doesn't have this notion of ordering of rows.

I like both of your suggestions!

[rxin]

@ueshin do you think we can add this in 0.5?

[ueshin]

If we limit the row access to .iloc[:10, ...] (head(10)) or .iloc[:, ...] (get all rows) as @thunterdb suggested, yes, we can do in 0.5.

[rxin]

Yes let's just do those two Tim suggested.

[ueshin]

I'll work on this.