Great_expectations: PandasExecutionEngine: pyarrow read_table() got an unexpected keyword argument 'compression'

[sfrancavilla]

Describe the bug
I'm trying to reproduce the same example described here but with .parquet file on S3 instead of .csv.

Basically test .yml configuration of a Datasource.

To Reproduce
Below you can find the code I'm trying to run:

import great_expectations as ge

context = ge.get_context()

config = """
class_name: Datasource
execution_engine:
    class_name: PandasExecutionEngine
data_connectors:
    my_custom_data_connector:
        class_name: InferredAssetS3DataConnector
        bucket: bucket_name
        prefix: prefix_name
        default_regex:
            pattern: (.+)\\.parquet
            group_names:
                - source_key_name
"""

context.test_yaml_config(config)

The code snippet correctly recognises partitions in the bucket folder but it's unable to load the sample data reference. Below the stacktrace of the issue:

Attempting to instantiate class from config...
        Instantiating as a Datasource, since class_name is Datasource
        Successfully instantiated Datasource
ExecutionEngine class name: PandasExecutionEngine
Data Connectors:
        my_custom_data_connector : InferredAssetS3DataConnector
        Available data_asset_names (1 of 1):
                DEFAULT_ASSET_NAME (1 of 1): ['ppr/downloaded_at=2021-02-24/data.parquet']
        Unmatched data_references (0 of 0): []
        Choosing an example data reference...
                Reference chosen: ppr/downloaded_at=2021-02-24/data.parquet
                Fetching batch data...
Traceback (most recent call last):
  File "scripts/ie/ppr/test_ge.py", line 53, in <module>
    context.test_yaml_config(config)
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 3162, in test_yaml_config
    raise e
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 3150, in test_yaml_config
    pretty_print=pretty_print
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/great_expectations/datasource/new_datasource.py", line 277, in self_check
    pretty_print=pretty_print, max_examples=max_examples
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/great_expectations/datasource/data_connector/data_connector.py", line 293, in self_check
    data_asset_name=data_asset_name,
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/great_expectations/datasource/data_connector/data_connector.py", line 330, in _self_check_fetch_batch
    batch_definition=batch_definition
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/great_expectations/datasource/data_connector/data_connector.py", line 95, in get_batch_data_and_metadata
    batch_spec=batch_spec
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/great_expectations/execution_engine/pandas_execution_engine.py", line 158, in get_batch_data_and_markers
    df = reader_fn(buf, **reader_options)
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/pandas/io/parquet.py", line 460, in read_parquet
    path, columns=columns, use_nullable_dtypes=use_nullable_dtypes, **kwargs
  File "/Users/stefanofrancavilla/src/env3.7.5/lib/python3.7/site-packages/pandas/io/parquet.py", line 222, in read
    path_or_handle, columns=columns, **kwargs
TypeError: read_table() got an unexpected keyword argument 'compression'

As you can see from the error, it seems read_table() does not accept a compression argument.

Expected behavior
The expected behavior would be the correct loading of the .parquet file and printing of the chosen sample data on stdout.

Environment (please complete the following information):

• Operating System: MacOs
• Great Expectations Version: 0.13.11
• pandas: 1.2.2
• pyarrow: 3.0.0

[eugmandel]

@sfrancavilla Thank you for creating this detailed report! We will triage the issue today.

[jcampbell]

@sfrancavilla - @eugmandel flagged this one for me. It looks like #2440 , which we recently merged to add support for the compression key is currently always adding the compression key. I've created a small PR to fix the issue: #2506

[sfrancavilla]

@jcampbell @eugmandel not sure if this will be solved soon but, I had some time today and wanted to give it a try on this issue as my first contribution but, happy to close my PR (#2557), and apologies if I missed any procedure in place. It seems the only tests failing are related to missing psycopg2 library - needed to dig a bit deeper on the reason why.

I kept the same tests as @jcampbell as reference.

[sfrancavilla]

@jcampbell @eugmandel just a quick update to let you know that all tests on my PR passed successfully.