Aws-data-wrangler: s3.to_parquet single file doesnt sanitize column names

[dwbelliston]

Describe the bug

According to docs the column names are suppose to be sanitized. https://github.com/awslabs/aws-data-wrangler/blob/master/awswrangler/s3.py#L871

When i write a single file to parquet, I am not seeing the column names be sanitized.

To Reproduce

python3.8
awswrangler==1.1.2

Code I am using:

        wr.s3.to_parquet(
            df=dataframe,
            path=f'{bucket}/{object_dir}/{object_name}',
            use_threads=False
        )

Input columns:

Index(['id', 'type', 'last_updated', 'name', 'creation_date', 'total.tags',
       'total.keywords', 'total.domains_tags', 'total.domains',
       'settings.device', 'settings.locality.geo_id', 'settings.locality.name',
       'settings.language', 'settings.country', 'settings.show',
       '_links.notes', '_links.keywords', '_links.tags', '_links.alerts',
       '_links.domains'],
      dtype='object')

Output py arrow table variable at https://github.com/awslabs/aws-data-wrangler/blob/4f962d5ad15c98e4ea296405a60d6e18d5b49e86/awswrangler/s3.py#L1189

table
pyarrow.Table
id: string
type: string
last_updated: int64
name: string
creation_date: int64
total.tags: int64
total.keywords: int64
total.domains_tags: int64
total.domains: int64
settings.device: string
settings.locality.geo_id: int64
settings.locality.name: string
settings.language: string
settings.country: string
settings.show: int64
_links.notes: string
_links.keywords: string
_links.tags: string
_links.alerts: string
_links.domains: string

[igorborgest]

Thanks @dwbelliston! Good catch!

I've just fixed in the commit above. And it is available on the development brach:

pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev

Will be officially released in the next version 1.2.0.

[dwbelliston]

🥳 - Super! Appreciate the work you are doing on the project.

[igorborgest]

Released on version 1.2.0