Aws-data-wrangler: Getting error ResourceNumberLimitExceededException) when calling the UpdateTable operation: Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE

[edvorkin]

P.S. Don't attach file. Please, prefer add code snippets directly in the message body.
I have the following code:
wr.s3.to_parquet(df=df, path=destination_path, dtype=types, dataset=True, database=DATABASE, table=TABLE, mode="append", catalog_versioning=False, partition_cols=['env','year','month','day','hour'])
I am executing this on S3 event when data file created in s3. At some point I got error "(ResourceNumberLimitExceededException) when calling the UpdateTable operation: Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE" . Is there any way not to update table version? The table stay the same, only partitions are added with new data.

[igorborgest]

Hi @edvorkin! Are you using what version?

[edvorkin]

@igorborgest I am using pip install awswrangler without specifying any version. I see in the path it is awswrangler-1.7.0.dist-info.

[igorborgest]

Hi @edvorkin!

I've just fixed it in the dev branch. Could you please test it before the official release?
The ideia is publish the fix in the version 1.8.0 this weekend.

pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev

[edvorkin]

Ok. Great I will test today. @igorborgest

[edvorkin]

@igorborgest tested. It does not throw the exception any more. Thanks

[igorborgest]

Thank you @edvorkin, it will be released on Sunday.

[igorborgest]

Hi @edvorkin !

It is totally off-topic, but we are stating a _"Who uses AWS Data Wrangler?"_ section. So feel free to add yourself if you want 😄 .

[igorborgest]

Released on version 1.8.0.