Aws-data-wrangler: Option to delete temp table in S3 when using CTAS

[sapientderek]

Is your idea related to a problem? Please describe.
I currently use wr.athena.read_sql_query() using ctas_approach=True (which is fantastically fast) in a daily batch process. Currently each time it runs the temp table's parquet files are stored in S3 which are often no longer needed. My current plan to delete these files from S3 is to either delete all temp tables with wr.s3.delete_objects('<S3-BUCKET>/temp') or create a lifecycle policy for the S3 bucket used.

Sometimes I want to keep certain temp tables for later debugging or re-downloading the table without repeating the Athena query, which these work arounds don't fix. Currently to find which temp table I want, I need to look at Athena's query history.

Describe the solution you'd like
wr.athena.read_sql_query() could have an optional argument to delete the temp table from S3 after reading. Alternatively (or in addition) the function could have an argument for custom temp table names so deletion/retention can be managed by the user.

Thanks for all the great work!

[igorborgest]

Thanks for the great ideas @sapientderek !

I've just added the keep_files and ctas_temp_table_name arguments to:

• wr.athena.read_sql_query()
• wr.athena.read_sql_table()

Will be available soon in the next release (1.1.0)!

[igorborgest]

Available on version 1.1.0

[sapientderek]

Thanks for all the great work @igorborgest !