Koalas: str.split(expand = True)

[mks2192]

in dataframe string split into multiple columns is not available. what is alternative to that

[itholic]

Would you mind if you show me a little bit of your code examples?

I'm not sure what you exactly want to, but maybe the below way will work for you??

>>> import databricks.koalas as ks
>>> from databricks.koalas.config import option_context

>>> kdf = ks.DataFrame({'col1': ['A_1', 'A_2', 'A_3', 'A_4', 'A_5']})
>>> kdf
  col1
0  A_1
1  A_2
2  A_3
3  A_4
4  A_5

>>> with option_context('compute.ops_on_diff_frames', True):
...     kdf['first_split'] = kdf.map_in_pandas(lambda pdf: pdf.col1.str.split('_').str[0].to_frame()).col1
...     kdf['second_split'] = kdf.map_in_pandas(lambda pdf: pdf.col1.str.split('_').str[1].to_frame()).col1
...
>>> kdf
  col1 first_split second_split
0  A_1           A            1
1  A_2           A            2
3  A_4           A            4
2  A_3           A            3
4  A_5           A            5

[mks2192]

Thanks. It was super helpful.

I believe its possible to use most of the pandas function using map_in_pandas ?

More specifically I was looking for this

Temp = kdf.map_in_pandas(lambda pdf:pdf.col1.str.split("_", expand = True))

Temp.columns = ['S1', 'S2']

Temp.head()

        S1     S2
0   A   1
1   A   2
3   A   4
2   A   3
4   A   5

[itholic]

@mks2192

I believe its possible to use most of the pandas function using map_in_pandas ?

It doesn't support for aggregation-like methods which needs operation through the whole dataset.

In this case, It's possible to use map_in_pandas because the str.split is applied to row by row.

If you wanna know about more detail, please check this documents.

It's really happy that It helped you :D

[ueshin]

I submitted a PR to support expand parameter with a couple of restrictions that:

• n parameter must be a positive integer
• the number of columns will always be n + 1.

Could you take a look at the PR #1432? Thanks!