Aws-data-wrangler: Unknown parameter in TableInput: "CatalogId", must be one of: Name, Description, Owner, LastAccessTime, LastAnalyzedTime, Retention, StorageDescriptor, PartitionKeys, ViewOriginalText, ViewExpandedText, TableType, Parameters, TargetTable

[qemtek]

Hey, while running this line in a larger pipeline:

wr.s3.to_parquet(df, path=f's3://{S3_BUCKET}/datasets/', dataset=True,
                 dtype=SCHEMA_COLUMNS, mode='overwrite' if full_refresh else 'append',
                 boto3_session=session, database=AWS_GLUE_DB, table=AWS_GLUE_TABLE,
                 partition_cols=['year'])

I get the following error:

Traceback (most recent call last):
  File "./src/upload_data_to_s3.py", line 88, in <module>
    upload_local_files_to_dataset(full_refresh=False)
  File "./src/upload_data_to_s3.py", line 83, in upload_local_files_to_dataset
    partition_cols=['year'])
  File "/home/circleci/project/venv/lib/python3.7/site-packages/awswrangler/s3.py", line 1217, in to_parquet
    projection_digits=projection_digits,
  File "/home/circleci/project/venv/lib/python3.7/site-packages/awswrangler/catalog.py", line 227, in create_parquet_table
    projection_digits=projection_digits,
  File "/home/circleci/project/venv/lib/python3.7/site-packages/awswrangler/catalog.py", line 1146, in _create_table
    upsert_table_parameters(parameters=parameters, database=database, table=table, boto3_session=session)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/awswrangler/catalog.py", line 1660, in upsert_table_parameters
    parameters=pars, database=database, table=table, catalog_id=catalog_id, boto3_session=session
  File "/home/circleci/project/venv/lib/python3.7/site-packages/awswrangler/catalog.py", line 1715, in overwrite_table_parameters
    client_glue.update_table(**args2)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/botocore/client.py", line 608, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/botocore/client.py", line 656, in _convert_to_request_dict
    api_params, operation_model)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/botocore/validate.py", line 297, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in TableInput: "CatalogId", must be one of: Name, Description, Owner, LastAccessTime, LastAnalyzedTime, Retention, StorageDescriptor, PartitionKeys, ViewOriginalText, ViewExpandedText, TableType, Parameters, TargetTable

Package versions:
Running

• Python 3.7
• Awswrangler 1.6.2
• Boto3 1.14.20
• Pandas 1.0.4

I get the same issue with previous versions of awswrangler (I tried 1.4 as well). I will try to make a reproducible example if I get time, but it looks like one of the input names has changed so maybe its an issue you will see straight away. This issue does not happen when I set mode='overwrite', only mode='append'.

[igorborgest]

Thanks @qemtek!

Seems that boto3 changed their response payload on version 1.14.18 for glue_client.get_table(). We will fix it.

[igorborgest]

Fix released on version 1.6.3.

[schot]

Thanks for the quick fix!

The breaking change seems to be in botocore 1.17.18, not boto3 per se. A workaround for those that can't immediately upgrade to 1.6.3 is to pin botocore to <1.17.8.

[igorborgest]

Hi @qemtek @schot !

It is totally off-topic, but we are stating a _"Who uses AWS Data Wrangler?"_ section. So feel free to add yourself if you want 😄 .