Aws-data-wrangler: Support for automatic index and timezone recovery from Parquet files.

[igorborgest]

When Parquet files are created by Wrangler/Pandas some metadata are stored into the file with hints about how to reconstruct Indexes correctly or how to localize a datetime column with a specific time zone.

• [X] When wr.s3.to_parquet(index=True) we want to preserve the current behaviour of materialize the index(es) as a parquet column. This is important because non materialised index(es) will not be detected by engines such Spark, PrestoDB, Redshift Spectrum, Athena, Hive, etc.
• [X] Support for index(es) recovery following Pandas metadata injected in the files.
• [X] Support for MultiIndex
• [X] Support for RangeIndex
• [X] Support for datetime timezone recovery.

Related commits:

• Add support for timezone and index for wr.s3.read_parquet() https://github.com/awslabs/aws-data-wrangler/commit/0434ec72c8096bc7112d1cda10b7c551080a1f6b
• Improve index recovery https://github.com/awslabs/aws-data-wrangler/commit/50b80aee13c5c53db644faf284d89aedcc6ab6e7
• Add support for multiindex recovery https://github.com/awslabs/aws-data-wrangler/commit/c64e7e0ccefeaa4814872d31a5ad8f7a7b3f3e69

Original discussion: https://github.com/awslabs/aws-data-wrangler/pull/339

cc: @alexifm @Digma

[igorborgest]

@alexifm @Digma

I've just added support for what we discussed previously.

Do you mind to test it from our development branch and check if it helps in your use case?
The ideia is to publish this enhancement in the version 1.9.0 later this week.

pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev

[alexifm]

Hey, I think this is awesome! I tested it out a bunch with different index types and partitions and I think the only round trip issues I found are known pyarrow issues, mostly relating to categoricals. Specific to what the wrangler is responsible for, I can't find any issues.

I did not test anything with the timezones.

[igorborgest]

Released on version 1.9.0

[Digma]

Thanks for releasing. I was off so I did not had time to test it but great that you went ahead and released that 👍

[Digma]

Tested it and everything seems to work ok for us. Thanks again!