Koalas: Py4JJavaError: An error occurred while calling o1446.filter.

[hanzigs]

Hi,
I have a proc_cnt which koalas df with column count, (coding in databricks cluster)

thres = 50
drop_list = list(set(proc_cnt.query('count >= @thres').index))
ks_df_drop = ks_df[ks_df.product_id.isin(drop_list)]

My query function throws ParseException

org.apache.spark.sql.catalyst.parser.ParseException: 
Py4JJavaError                             Traceback (most recent call last)
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:

Py4JJavaError: An error occurred while calling o1446.filter.
: org.apache.spark.sql.catalyst.parser.ParseException: 
extraneous input '@' expecting {'(', 'SELECT', 'FROM...………...

May I know the issue
Thanks

[HyukjinKwon]

Can you check the documentation of https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.query.html? We don't currently support @ properly. There is a workaround documented there as well.

[hanzigs]

Thanks for that, removed @ and used as it is. worked.
and
Couldn't find set() in koalas, how to create set?

[HyukjinKwon]

Can you point out the pandas API link? I would like to check.

[hanzigs]

Thanks, just the python data structure set()

drop_list = list(set(proc_cnt.query('count >= 50').index))

The set() will create the iterable values from the index values.

[HyukjinKwon]

Oh, you mean the Python built-in function set, right? It wouldn't work out of the box. You might want to check https://github.com/databricks/koalas/pull/1420