Aws-data-wrangler: Add table properties for athena create csv

[kylepierce]

Is your idea related to a problem? Please describe.
I am trying to automate my athena table creation for multiple environments. CSV are loaded into s3 unedited. When I create the table with the UI it detects a header row. With this package it doesnt.

Describe the solution you'd like
Add TBLPROPERTIES like skip.header.line.count"="1 to create_csv_table

[igorborgest]

Hi @kylepierce, thank you for reach out.

Could you paste here the code snippet where you are trying do create it?

[kylepierce]

I dont see it in the boto3 docs so maybe its not supported at the moment.

Ideally it would be a dict that can accept other table properties. (last line)

wr.catalog.create_csv_table( database='lake', table='table', path='path', columns_types=data, compression='gzip', description='My own table!', parameters={'source': 'postgresql'}, table_properties={'skip.header.line.count': "1"} )

[kylepierce]

data refrences a dict

{ "name": "varchar(100)", "clicks": "bigint", "sum": "double" }

[igorborgest]

@kylepierce!

I've already added the skip_header_line_count as a new argument to our development branch. Do you mind to test it?
The ideia is release it in the next version 1.7.0 this week.

pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev

_P.S. For details see the related commit.
P.P.S. For the documentation related to the development branch see.
P.P.P.S By now it only support the LazySimpleSerDe (And not the OpenCSVSerDe), so quoted values will work._

[kylepierce]

Wow thanks! Works as a test.

[kylepierce]

Looking at your P.P.P.S is there any way to specify which SerDe I want to use or because it is a csv I am locked into "LazySimpleSerDe"?

I have a table that has every value quoted. I see that one of the parameters is "areColumnsQuoted" and that marked false as default.

Maybe I am trying to use this package too much and should look at just using DDL creation directly.

[igorborgest]

Released in 1.7.0!

[igorborgest]

@kylepierce about the OpenCSVSerDe support, if you still want it, could you please open a separated issue for that?

[igorborgest]

Hi @kylepierce !

It is totally off-topic, but we are stating a _"Who uses AWS Data Wrangler?"_ section. So feel free to add yourself if you want 😄 .