Aws-data-wrangler: pandas.to_parquet writes out parquet files with data types not supported by athena/glue
session.pandas.to_parquet writes out parquet files with data types not support by athena/glue, which results in things like HIVE_BAD_DATA: Field primary_key's type INT64 in parquet is incompatible with type string defined in table schema
jasonamyers
All 4 comments
Hi @jasonamyers! Thanks for open this.
Do you have more context? I would like to reproduce your issue here in my environment.
igorborgest
on 6 Nov 2019
👍1
I'll make a good test case and post it back.
jasonamyers
on 6 Nov 2019
🚀1
@jasonamyers thank you very much!
igorborgest
on 6 Nov 2019
I've been able to declare better types in my dataframe and get those to go all the way through to athena properly, so I'm going to close this issue for now. I just expected that this would take care of that for me, which wasn't a good expectation on my part. Thank you for this library!
jasonamyers
on 7 Nov 2019
👍2
Was this page helpful?
0 / 5 - 0 ratings
Related issues
bryanyang0528
路
3Comments
martinSpears-ECS
路
3Comments
radcheb
路
4Comments
dwbelliston
路
3Comments
bryanyang0528
路
5Comments
Most helpful comment
I've been able to declare better types in my dataframe and get those to go all the way through to athena properly, so I'm going to close this issue for now. I just expected that this would take care of that for me, which wasn't a good expectation on my part. Thank you for this library!