Aws-data-wrangler: 'Could not convert <NA> with type NAType' when writing pandas DF to S3 parquet.

[andymcarter]

When trying to write a pandas dataframe to S3 in parquet format, I receive this error message.

[ERROR] ArrowInvalid: ('Could not convert with type NAType: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column ic_icp_id with type object')

It seems related to https://github.com/awslabs/aws-data-wrangler/issues/132 except in that case, it was writing to redshift.

In this case the column is only needed as an intermediate key to join another dataframe; I can get around the issue by dropping the column before writing out to S3. But doing this for each instance of obj with NAs feels like an incorrect approach.

To Reproduce
python 3.7
wr.__version__ = 1.3.0

[igorborgest]

Hi @andymcarter! Thanks for reaching out.

Wrangler can't infer data type from Pandas columns of type "object" fully of NAs values. You need to have a defined data type or at least one value to be inferred.

Based on the column name "ic_icp_id" I would suggest you to cast this column to Int64 type for example. (Attention to the capital "I")

df["ic_icp_id"] = df["ic_icp_id"].astype("Int64")

What do you think?

[andymcarter]

Thanks for the suggestion. If I have understood, you think the issue is that the entire column is NA? I have checked this column and only around 1% of values are NA, the rest are correct Int64s.

EDIT: On further investigation, I was performing a pandas join on this column, and setting an Int64 column as the index on the RHS of the join had coerced the dtype on the LHS to object. More investigation needed on my part here I think. Feel free to close.

[igorborgest]

Hi @andymcarter !

It is totally off-topic, but we are stating a _"Who uses AWS Data Wrangler?"_ section. So feel free to add yourself if you want 😄 .