Koalas: Triage Top Missing APIs to explicitly don't support or implement.

[HyukjinKwon]

@ueshin wrote and ran some notebooks to analyze statistics from missing API calls as below. We might need to explicitly don't support with a proper workaround in its exception message and/or implement it first.

• [x] Series.__iter__(self)
• [x] DataFrame.apply(self, func, axis=0, broadcast=None, raw=False, reduce=None, result_type=None, args=(), **kwds) #1259
• [x] Series.values
• [x] Index.values
• [x] DataFrame.info(self, verbose=None, buf=None, max_cols=None, memory_usage=None, null_counts=None) - https://github.com/databricks/koalas/issues/872
• [x] Index.to_numpy(self, dtype=None, copy=False)
• [x] DataFrame.iterrows(self)
• [x] DataFrame.unstack(self, level=-1, fill_value=None) #1295
• [x] DataFrame.rename(self, mapper=None, index=None, columns=None, axis=None, copy=True, inplace=False, level=None)
• [ ] DataFrame.tail(self, n=5)
• [x] Index.__iter__(self)
• [x] DataFrame.where(self, cond, other=nan, inplace=False, axis=None, level=None, errors='raise', try_cast=False, raise_on_error=None)
• [x] DataFrame.values
• [x] DataFrameGroupBy.head(self, n=5)
• [x] DataFrame.query(self, expr, inplace=False, **kwargs)
• [x] DataFrame.take(self, indices, axis=0, is_copy=True, **kwargs) #1292
• [x] DataFrame.take(self, indices, axis=0, convert=None, is_copy=True, **kwargs) #1292
• [x] Index.tolist(self)
• [ ] SeriesGroupBy.unique
• [x] Series.keys
• [x] Series.replace

[itholic]

i think Series.__iter__(self) is quite widely & frequently used by data scientists.

so in my opinion, how about make it available for now with proper warning message although it has risk of memory issue when large-size dataset?

now it raises PandasNotImplementedError like below,

>>> for a in kser:
...     a
...
Traceback (most recent call last):
databricks.koalas.exceptions.PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.

how about just use to_numpy() internally and explicitly notice to the doc about the risk when using this function. (actually this seems need to not only Series.__iter__, but other functions that has similar risk like this)

Or, i think we may consider to make the new option like large_data_calculation to make possible execute these functions without exception. (default False, maybe)

[HyukjinKwon]

Spark has toLocalIterator so we might be able to use it for iteritems or iterrows.

For __iter__, it's really easy for users to make mistakes (e.g., they could even just do list(ser) if we implement __iter__.

[HyukjinKwon]

Think about such case that users run a for loop expecting a lazy execution magically, e.g.:

for data for ser:
    ....

and it suddenly hangs or out-of-memory error.
Their codes are usually from pandas so it looks to me too much error prone

I think we can maybe implement both iteritems or iterrows with explicit warnings for now .. but __iter__ I am not sure yet.

[itholic]

@HyukjinKwon

thanks for sharing your experiences !! okay now i've got totally understood.

for now, i seems better to just keep as an open issue to discuss.

[HyukjinKwon]

@itholic, when you have some time, can you take a look for DataFrame.take, and DataFrame.query?

take seems we can just reuse existing indexer, and query seems we can just delegate to Spark's filter API. For example,kdf.to_spark().filter("...").

[HyukjinKwon]

SeriesGroupBy.unique, seems easy as well but it seems it returns an array. We can either explicitly don't support or support this by returning a series.

[itholic]

@HyukjinKwon sure, i'll manage them after finishing pandas 1.0.0 support for Koalas 😸
(#1197 )

[itholic]

@HyukjinKwon

SeriesGroupBy.unique, seems easy as well but it seems it returns an array. We can either explicitly don't support or support this by returning a series.

I have investigated SeriesGroupBy.unique, pandas seems to return a Series , not an array like the below.

>>> pdf
   a  b
0  1  4
1  2  5
2  3  6

>>> pdf.groupby(['a'])['b'].unique()
a
1    [4]
2    [5]
3    [6]
Name: b, dtype: object

so, i think we can implement SeriesGroupBy.unique just same with pandas' one.

[HyukjinKwon]

Yes, I think it's fine to implement.

[itholic]

Okay let me implement, thanks :)

[itholic]

i've investigated DataFrame.take.

take seems we can just reuse existing indexer

pandas can just do like the below with using iLocIndexer.

>>> pdf
     name   class  max_speed
0  falcon    bird      389.0
2  parrot    bird       24.0
3    lion  mammal       80.5
1  monkey  mammal        NaN

>>> pdf.take([0, 3])
     name   class  max_speed
0  falcon    bird      389.0
1  monkey  mammal        NaN

>>> pdf.iloc[[0, 3], :]
     name   class  max_speed
0  falcon    bird      389.0
1  monkey  mammal        NaN

but in Koalas, we're raising Error like the below.

>>> kdf
     name   class  max_speed
0  falcon    bird      389.0
2  parrot    bird       24.0
3    lion  mammal       80.5
1  monkey  mammal        NaN

>>> kdf.iloc[[0, 3], :]
Traceback (most recent call last):
...
databricks.koalas.exceptions.SparkPandasNotImplementedError: .iloc requires numeric slice or conditional boolean Index, got <class 'list'> You are trying to use pandas function .iloc[..., ...], use spark function select, where

So, i think maybe we can handle this via similar way like the below?

>>> kdf.take([0, 3])
Traceback (most recent call last):
...
databricks.koalas.exceptions.PandasNotImplementedError: The function `ks.DataFrame.take()` is not implemented, use 'to_pandas().DataFrame.take()' instead if you sure about the size of the data is not so big enough.

[HyukjinKwon]

Actually we will be able to work around ^ case. I am working on map_partitions. We will be able to use it to implement take.

[HyukjinKwon]

I am hesitant about implementing DataFrame.unstack(self, level=-1, fill_value=None) case (see https://github.com/databricks/koalas/issues/886)

[itholic]

Actually we will be able to work around ^ case. I am working on map_partitions. We will be able to use it to implement take.

Ah, i just explored that API. please let me know if you finish implementing map_partitions !! 👍

[itholic]

I am hesitant about implementing DataFrame.unstack(self, level=-1, fill_value=None) case (see #886)

I'm not pretty sure but i think we'd better implement that even with limited functionality, if there are many user's requirements or we can utilize them for implementing of other APIs.

[HyukjinKwon]

^ The problem is Spark doesn't really scale out in columns. If there's 10 columns and 10,000 rows, it becomes 100,000 columns, and Spark won't probably work. Let's see if people actually request this or not.

[HyukjinKwon]

DataFrame.tail will be tracked at #343. SeriesGroupBy.unique will be tracked at #1320