Aws-data-wrangler: wr.s3.to_parquet(table=table) create table name different than specified

[mechatroNick]

table = "purchase-activity-topic-test-glue-table"
response = wr.s3.to_parquet(
    df=df,
    path=path,
    dataset=True,
    database=database,
    table=table,
    mode=mode,
    partition_cols=["year", "month", "day"]
)

What you would expect is for Glue:CreateTable is purchase-activity-topic-test-glue-table
What you actually get when Glue:CreateTable is purchase_activity_topic_test_glue_table

The subsequent times when you try to write data, Glue:GetTable will look for purchase-activity-topic-test-glue-table.

Even though you successfully wrote data to Glue::Table purchase_activity_topic_test_glue_table, now subsequent write will fail because purchase-activity-topic-test-glue-table does not exist.

Very strange things can occur when you automatically convert - into _

This took me a few round trips to CloudTrail and back because I overlooked the Table's name :(

AWS Console:

My EKS Helm Chart config:

[igorborgest]

Hi @mechatroNick , thanks for reaching out!

It is not a bug, it is a feature that will sanitize your table and columns names that are not supported on Amazon Athena and others engines (Reference).

Generally speaking I recommend you to avoid hyphens in tables and columns names to avoid headaches, underscores are much more interchangeable among databases/engines.

P.S. You can also use the wr.catalog.sanitize_table_name() as a helper :slightly_smiling_face:

[mechatroNick]

@igorborgest thank you! could you please note that Athena is hyphens sensitive in the README?