Aws-data-wrangler: Error with wr.s3.store_parquet_metadata: ValueError: number sections must be larger than 0.

[qemtek]

Hi, I'm using the latest version of awswrangler (great package by the way). I am uploading a parquet dataset using the wr.s3.to_parquet() function, with no partition. For some reason, one of my datasets dosent update in my AWS Glue dashboard, so I have to run the crawler (dont know if it is supposed to update or not).

When I run the crawler, the error seems to appear because I dont have a partition in the dataset? I'm not too sure. I'm using Python 3.7 with Pycharm and the latest mac OS Catalina 10.15.3.

Ok a side note, do I have to run the crawler to update the table info, so I can run queries in Athena? It seems to take a long time (30min+) to crawl my dataset of around 1 million rows. I'm not too sure on the optimal way to add data to the dataset each day, or if I can implement that solution using awswrangler.

Anyway,

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/shape_base.py", line 770, in array_split
Nsections = len(indices_or_sections) + 1
TypeError: object of type 'int' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/chriscollins/Documents/GitHub/RPScraper/RPScraper/src/run_glue_crawler.py", line 26, in
boto3_session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/awswrangler/s3.py", line 1949, in store_parquet_metadata
boto3_session=session,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/awswrangler/catalog.py", line 247, in add_parquet_partitions
_add_partitions(database=database, table=table, boto3_session=boto3_session, inputs=inputs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/awswrangler/catalog.py", line 1089, in _add_partitions
chunks: List[List[Dict[str, Any]]] = _utils.chunkify(lst=inputs, max_length=100)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/awswrangler/_utils.py", line 128, in chunkify
np_chunks = np.array_split(lst, n)
File "<__array_function__ internals>", line 6, in array_split
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/shape_base.py", line 776, in array_split
raise ValueError('number sections must be larger than 0.')
ValueError: number sections must be larger than 0.

Process finished with exit code 1

[qemtek]

Also I will add, it looks like the crawler completes something.. Because the modified date on the table in AWS Glue is updated and I can query the data

[qemtek]

Sorry, the command I am running is here, where the path is a folder of snappy compressed parquet files generated using the to_parquet function.

res = wr.s3.store_parquet_metadata(
    path=f"s3://{S3_BUCKET}/datasets/",
    database=AWS_GLUE_DB,
    table=AWS_GLUE_TABLE,
    dataset=True,
    use_threads=True,
    mode=mode,
    boto3_session=session
)

[igorborgest]

Hey @qemtek! Thanks a lot for reporting it, I will troubleshoot!

[igorborgest]

@qemtek as you said looks like that it is related to non-partitioned datasets, I will fix it.

Now about your process itself:

At least based on what I understood about your flow, you don't need to use the wr.s3.store_parquet_metadata() because the wr.s3.to_parquet() also can create the table metadata on Glue Catalog automatically. So, in your case a wr.s3.to_parquet(mode="append") should be enough to add new data to your table and be reflected on Athena queries.

Take a look on these tutorial it could help you. All the examples into it could be reflected on Athena passing database and table arguments.

Let me know what you think!

[qemtek]

Thanks! For some reason, the metadata wasnt being updated when I was using to_parquet, but only on one of my tables. I've added partitioning to both tables now because I think its good practise and the meta data for all my tables updates automatically now, which is awesome!

[igorborgest]

Thanks @qemtek.

The original bug was fixed on the development branch available through:

pip install git+https://github.com/awslabs/aws-data-wrangler.git@dev

Any feedback are welcome.

Will be officially available on next version 1.2.0. Thanks!

[igorborgest]

Released on version 1.2.0