Koalas: Filtering rows throws error when result is empty

[brookewenig]

In Pandas, if I filter for rows and there are none that satisfy the condition, it returns an empty DataFrame.

With Koalas, it throws an error. I think it should return an empty dataframe, like Pandas.

Note: data is a koalas dataframe in the screenshot below

[itholic]

Thanks for your issue reporting!

Sorry, but if you don't mind, could you show me the python code you used to make Koalas DataFrame if it possible?

I just tested it with the example you uploaded.

In my case, both results are returning an Empty DataFrame like below:

Thanks for your reporting again, Have a nice day :)

[brookewenig]

When going from a Pandas DF to Koalas DF, I agree, there are no issues. However, this issue seems to come up when reading directly from a CSV file. I can reproduce this issue with two different CSV sources. Take a look at the notebook linked here and you can see the error: https://databricks-prod-cloudfront.cloud.databricks.com/public/793177bc53e528530b06c78a4fa0e086/0/15446625/100107/latest.html

[ueshin]

Hmm, that's interesting. Let me check, too. Thanks for sharing the notebook!

[harupy]

I wrote a simple notebook to investigate this issue.

https://github.com/harupy/temp/blob/master/koalas_filter_issue.ipynb

Seems like the error occurs when pyarrow converts a datetime column.

[HyukjinKwon]

Thanks @harupy and @itholic for analysis and the codes!