Aws-data-wrangler: validate_schema parameter via pyarrow.parquet.ParquetDataset

[ghost]

Could you please add the parameter "validation_schema" to the wr.s3.read_parquet module?

[igorborgest]

Hi @jewelltp, could you please explain what is the use case? Do you mean pass it directly to pyarrow.parquet.ParquetDataset?

Today we keep the default True.

[ghost]

@igorborgest I am running into issues where the schemas between separate files is different. This seems to be an issue with the way my company had set up the files. To bypass this, I can use validate_schema=False.

[ghost]

@igorborgest - yes, just want to pass this to wr.s3.read_parquet which would then pass it to pyarrow.parquet.ParquetDataset

[igorborgest]

@jewelltp Ok, we will expose this argument in the next version 1.0.2 that will be released in the next two or three days.

Thanks for the suggestion!

[ghost]

@igorborgest thanks! Just started using this module a couple days ago and really like what I see. Great Job. Looking forward to how this package evolves!

[ghost]

@igorborgest pull request sent to address issue.

[ghost]

@igorborgest merged. Will submit tests next time, sorry. This is my first fork and contribution on Github, so still learning.

[igorborgest]

Thanks @jewelltp, this issue is resolved on version 1.0.2. Please, give feedback when possible.