Aws-data-wrangler: Support parquet logical type of MAP

[robertaves]

Looking to get support for outputting Tuple[str, str] or non nested dictionary's (key value pairs ) outputted logical dt of MAP in parquet file.

I understand dependent libraries do not yet support it by looking at this pull https://github.com/apache/arrow/pull/5774. Looks like it might be wrapping up development soon, does it make sense to eventually support?

Thoughts?

[igorborgest]

Thanks for the contribution @robertaves, this makes total sense.

Let's keep following this arrow's PR to develop our side when it is possible.

[sujayramaiah]

@igorborgest Looks like the arrow's PR has been closed.

[robertaves]

looks like pyarrow 0.16 added support for maparray. Can this be unblocked? would be great for athena maps

[igorborgest]

Thanks @robertaves! Let's work on that!

[igorborgest]

We will add support for nested types on version 1.0.0. Coming soon!

[igorborgest]

Hi! We will release AWS Data Wrangler 1.0.0 on April 10 and we are collecting as much feedback as possible.

This issue should be resolved with this release. So, please test it if possible, and any feedback about the docs, the library or this specific functionality will be very welcome.

• Install: pip install git+https://github.com/awslabs/[email protected]
• Docs: https://aws-data-wrangler.readthedocs.io/en/dev-1.0.0/
• Code: https://github.com/awslabs/aws-data-wrangler/tree/dev-1.0.0

[igorborgest]

Thank you for the contribution. Please, reopen the issue if it persists even after the version 1.0.0.