Koalas: Is there something wrong with concat when axis=1?

[Tom-Deng]

tmp3.shape is expected to be (195638, 32)

dtype info

[HyukjinKwon]

@Tom-Deng, do you mind showing the results of concat(tmp_list[0].head(10), tmp_list[1].head(10))? It will be helpful to show tmp_list[0].head(10) and tmp_list[1].head(10).

[Tom-Deng]

tmp_list[0].head(10)

tmp_list[1].head(10)

csv file
csv_file.zip

[HyukjinKwon]

Can you show your full codes? It produces the same results in my side:

import databricks.koalas as ks
tmp0 = ks.read_csv("tmp_list_0.csv")
tmp1 = ks.read_csv("tmp_list_1.csv")
ks.concat([tmp0, tmp1], axis=1).shape

(43018, 32)



md5-90f3aaed5d6708adc978aeb7d45b0158

(43018, 32)
```

[Tom-Deng]

reference: https://github.com/szilard/benchm-ml

The wrong place:

full code
github.tar.gz

version info

[HyukjinKwon]

@Tom-Deng, can you post a copy-and-paste-able codes? It's a bit difficult for me to guess and write the codes to reproduce your case.

[Tom-Deng]

Code and used data are in compressed package "github.tar.gz" with last comment.

[ueshin]

I found out the reason. It's because you created the d1 by d1 = d1a.append(d1b) which will contain duplicated index values.
So far Koalas uses join on index columns to implement concat(..., axis=1), so if there are duplicated values, it will produce more rows than the original row number.

Could you use d1 = d1a.append(d1b, ignore_index=True) as a workaround?

[Tom-Deng]

I found out the reason. It's because you created the d1 by d1 = d1a.append(d1b) which will contain duplicated index values.
So far Koalas uses join on index columns to implement concat(..., axis=1), so if there are duplicated values, it will produce more rows than the original row number.

Could you use d1 = d1a.append(d1b, ignore_index=True) as a workaround?

This modification is valid.